vulkan: use density gate for MUL_MAT_VEC_ID path - #27332
Conversation
|
I ran this on my 5090 using the command line from the issue, and while it is marginally better for B==9, it is slower for larger values. So this would need more tuning: |
84b3b8f to
aba6a48
Compare
|
I have added For now, not quite sure how to proceed in this path, because I don't have a 5090 to tune blackwell but I tried the gated fix in my latest revision on my ampere cards (3090 + a4000) and don't see any degradation, I am fairly certain in between AMD cards there could also be the same blackwell/ampere batch processing discrepencies where different cards might require different tuning 
|
|
I doubt it's related to architectural differences. I think the coopmat2 path does better with small batches than coopmat1 due to the |
github-actions
Bot
added
Vulkan
This comment was marked as low quality.
This comment was marked as low quality.
Replace the fixed 8-token cutoff with the ggml-org#25356 density gate (n_tokens * experts_per_token <= 2 * n_experts, cap 64 tokens) on devices without coopmat2. The tiled path is slow at small batch on coopmat1/no-coopmat hardware (decode cliff at 9+ concurrent sequences); validated on gfx1013, gfx1100, gfx1151 and NV coopmat1. coopmat2 devices (Blackwell) keep the fixed cutoff - their tiled path handles small batches well. Assisted-by: DeepSeek V4 Flash
aba6a48 to
4dfa152
Compare
|
@jeffbolznv thank you, you were right, it wasn't hardware differences, it ended up being based on coopmatmul class based, so I've dropped the gate on AMD class and gone specifically to Open question is whether the gate should be the long term fix or whether |
|
I did some experimentation today. It's not just the enable_smaller_matrices optimization, there is also some tile size and tile size selection tuning needed (and these are the bulk of the perf gain). Here are some measurements, all on RTX 5090 with coopmat2 disabled: I don't object to the current PR. If somebody has the time and AMD/Intel hardware available, it would be good to tune the tiling parameters and then maybe revisit this. It should also benefit larger prompt processing workloads. It doesn't make sense to tune it based on NVIDIA hardware performance. |
|
I couldnt measure any improvements on 2x R9700, Q3.8-27B and Q3.6 35B with single / parallel sessions up to np=3 with MTP=1 (because otherwise it trashes parallel session performance). |
|
Independent Linux Strix Halo / RADV test of current head System: Ryzen AI MAX+ 395 / Radeon 8060S ( Model: Build: Release + Mean ± sample SD, aggregate decode throughput:
Prompt processing was neutral at every B (all deltas between -0.16% and +0.36%). Targeted correctness at the exact PR head: So on Linux/RADV Strix Halo this head removes most of the B=8→9 MoE decode cliff without a measurable B≤8 or prompt-processing regression. |
|
@jeffbolznv Thank you for the coopmat1 experiments, the B=9 collapse on the 5090 with coopmat2 disabled was instrumental to validate the gate fixes, your PR-vs-master table (B=9 +40%, B=32 +4%) confirms the gate helps on the coopmat1 path too, not just for AMD devices. Agreed that tile-size tuning + enable_smaller_matrices is the better root-cause fix for coopmat1 since your numbers beat the gate at every B>=9 and also lift prefill. That portion of the work ideally requires AMD/Intel hardware to tune and likely doesnt benefit from just Nvidia hardware performance tuning. One context note from my side: My hardware (BC-250, gfx1013) has no matrix cores at all, no coopmat1, no coopmat2, so I'm unable to tile tune from here. The gate is the only applicable fix for my class of hardware. My data (B=9 +36%, B=16 +27%, B=64 +21%, B<=8 neutral) stands as the no-matrix-core case for the gate. Separately, I have Ampere cards (3090 + A4000, no coopmat2) that could validate the gate on another no-coopmat2 class via Vulkan. The tiling work itself is best done in a separate PR with AMD/Intel hardware, as you said: NVIDIA-derived tile parameters won't port. If the tiling work later supersedes the gate, the single !coopmat2 condition keeps the revert to a one-liner. @Stoney49th Thanks for testing on RDNA4. Worth noting: the gate is intentionally inactive on coopmat2-class devices (RDNA4 included) since they keep the fixed 8-token cutoff, since the tiled path handles small batches well there. So no improvement is the expected result on that hardware, particularly thank you for confirming and thanks for confirming it doesn't regress either. If you notice B=9 decode drop on RDNA4, that would be a seperate issue worth reporting @frizikk Thanks for the test, this is the cleanest validation: interleaved parent/head runs on the exact hardware from #25356, tight SDs, B=9 +52.6% and B<=8/prefill within noise. The 872/872 MUL_MAT_ID test on the head is appreciated too. |
4 participants
Replace the fixed 8-token cutoff with the #25356 density gate (
n_tokens * experts_per_token <= 2 * n_experts, capped at 64 tokens). Avoids the batch-9 decode regression on AMD RADV; validated on gfx1151, RDNA3 and gfx1013 (BC-250): +36% at B=9, +27% at B=16, +21% at B=64, neutral at B<=8.Assisted-by: DeepSeek V4 Flash
Overview
Replace the hardcoded
batch <= 8threshold for the VulkanGGML_OP_MUL_MAT_IDMMV path with the routed-density heuristic proposed in #25356.For MoE decode, the MMV kernel performs better for small routed workloads, while the tiled kernel wins at larger workloads. The fixed cutoff causes a sharp kernel-selection regression when going from 8 to 9 concurrent sequences on AMD RADV.
The new gate keeps MMV selected while:
and
n_tokens <= 64.The existing F32/F16/quantized type check is unchanged, and large-batch prefill remains on the tiled path.
Additional information
On a BC-250 (
gfx1013), the new gate produced:The same density heuristic has previously been validated on Strix Halo (
gfx1151) and RDNA3 in the discussion of #25356.Forcing the tiled path everywhere was also tested on
gfx1013and performed substantially worse (-44% at tg64, about -50% at B=1), supporting kernel selection rather than removal of the MMV path.The change is limited to
ggml_vk_use_mul_mat_vec_id(): 9 insertions, 1 deletion, with no new configuration or environment knobs.Correctness:
test-backend-ops -b Vulkan0 -o MUL_MAT_IDpasses872/872on the patched build (gfx1013, RADV).
Related work
Repro
Reproduction (llama-batched-bench, Qwen3.5-35B-A3B, 4-node RPC):
stock: B=8 52.6, B=9 35.9 t/s (cliff)
patched: B=8 51.5, B=9 48.2 t/s (+36%)
Requirements